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Appendix A. Required Elements of 
AO and Theory of Change 


Nonnegotiable Elements of the AO Grant 


1. 


8. 


Explicit articulation of two or more educational pathways, linked to career pathways, that begin 
with adult basic education or English as a second language classes and continue to a college- 
level certificate and beyond 


Evidence of strong local demand for the selected pathways, including the presence on the 
workforce investment board demand list for the local area or other local data demonstrating 
robust demand 


Acceleration strategies, including contextualized learning and the use of hybrid (online and 
classroom-based) course designs 


Evidence-based dual enrollment strategies, including paired courses and I-BEST and I-BEST- 
like approaches 


Comprehensive academic and social student supports (e.g., tutoring, child care, transportation, 
access to public benefits, and subsidized jobs) 


Achievement of marketable, stackable, credit-bearing certificates and degrees and college 
readiness, with an explicit goal of bypassing developmental education 


Award of some college-level professional-technical credits, which must be transcripted the 
quarter or semester in which they are earned 


Partnerships with workforce investment boards and employers 


(Source: JFF 2011) 


States and colleges were expected to adhere to the nonnegotiable elements of the model except 


where infeasible. The elements specify that the states’ programs should offer career pathways that are 


at least 12 credit-hours long, at least two pathways should be established in each of at least eight 


colleges, and pathways should have at least 25 percent team teaching. Students eligible for AO must 


have fallen within NRS functioning levels 4-6 (6th to 12th grade levels) on math, reading, or writing, or 


NRS functioning levels 5-6 in English-language skills. Enrolled students may have had a secondary 


school credential as long as they fell within the eligible skill ranges. States were asked to identify policy 


APPENDIX A | 


levers and were expected to make at least 80 percent progress toward their policy goals by the end of 
the grant period. The goal is that within three years of operation, each participating state would 
produce at least 3,600 credentials. States were to offer credentials in industries with sufficient labor 


demand so students could reasonably become employed within their areas of study. 


Figure A.1 shows JFF’s original theory of change for the AO initiative, originally called “ABE to 


Credentials.” 
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FIGURE A.1 


Accelrating Opportunity Theory of Change 


Stakeholders 
& Levers 


Stakeholders 


Key beneficiaries, 
implementing or 
enabling entities, 
supporters, and 
funders with a 
demonstrable interest 
in the outcomes: 
community/technical 
colleges, ABE 
programs, higher 
education agencies & 
design teams, state 
policymakers, federal 
agencies, CBOs, WIBs, 
employers, students, 
TA providers, 
philanthropic partners. 


Levers 

Approaches that can be 
taken or domains that 
can be acted on to 
change behaviors, 
conditions, or 
attitudes. 
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; Scale & Sustainability 


Activities & Interim Outcomes 


College & Career Pathways 


Evidence-based instructional and programmatic models that promote transition to and 
completion of credentialing programs in high-demand fields. Pathways must include 
acceleration and dual-enrollment strategies and comprehensive support, and culminate in 


marketable credentials and college readiness. 


Culture Shift Ee 


The necessary changes in attitude at community colleges, and among policymakers, 
employers, and ABE students themselves to view those students as valued members of the 
community college population capable of earning marketable credentials and beyond, and 
worthy of governmental funding. 


The increased percentage of student participation in ABE to Credentials pathways within a set 
of colleges in multiple states (depth), and the spread of pathway innovations to additional 
colleges and additional states (breadth); the ability for the innovations to continue over time as 


culture, environment, and postsecondary systems of multiple states. 


evidenced by viable funding mechanisms and the embeddedness of the innovations in the ! 
| 


Accelerating Opportunity Theory of Change — Definitions 


Long-Term Goals 


System 
Governing or 
coordinating higher 


| education state 


agencies focused on 
community colleges 
and ABE programs, 
public higher 
education institutions, 
related state agencies 
(e.g., workforce 
development, labor, 
commerce), 
associated employers, 
and community-based 
providers working in 
concert to develop 
labor-market-ready 
adults with marketable 
postsecondary 
credentials. 


| Students 


Low-skill adult learners 
in community college- 
based ABE, ESL, and 
ASE programs 


Accelerating Opportunity Theory of Change — Path to Impact for System and Students 


Stakeholders Activities 


& Levers 


2-Year Outcomes 


College & Career Pathways 


Community/Technical Colleges # ABE Programs @ State Policymakers Higher Education Agencies 


Federal Agencies @ WIBs & One-Stops ¢ Employers ¢ Community-Based Organizations ¢ Students 


Technical Assistance Providers ¢ Philanthropic Partners 


= Target faculty professional 
development 

= Engage faculty in course & 
pathway design, team teaching 

= Redesign curriculum 

= Create learning networks 

= Analyze labor market & engage 
employers 


Culture Shift 


= Engage champions 
= Launch strategic 


Employers actively engage with 
colleges on pathway development 
2 viable I-BEST or I-BEST-like 
pathways per participating college 
Multiple faculty members per 
college willing & able to deliver 
integrated curriculum to ABE 
students 

Implementation reflects emerging 
evidence & innovation 


Growing awareness of 
problem/solutions by colleges, 
employers, states 

Greater ABE student access to 


Data communications campus resources 
Program = Track data on ABE students Aue suiclents SR as Jalen 
=» Promote access to campus population/pipeline in institutions 
Innovation recente Far aurlaas ABE students seek college & career 
pathways 
- States, colleges invest in ABE data 
Policy tracking 
Financing 


Scale & Sustainability 


= Identify multi-sector resources 

= Identify & advocate for removal 
of policy barriers in all sectors 

= Provide state tools for cost- 
benefit analysis 


Some financial aid barriers 
removed; = 2 states successfully 
using models for ABE students to 
access Pell grants 

Colleges & students gain access to 
untapped state, federal & employer 
funding to support pathways 
Capacity, tools & data available in 
states & colleges to conduct cost- 
benefit analysis 

Colleges and states have greater 
capacity to track ABE student 
progress/outcomes & to link data 
to labor market 


Source: JFF (2011); reproduced here with permission. 


4-Year Outcomes 


Multiple transparent ABE-to- 
credential pathways aligned with 
labor market in at least 8 
colleges/state 

Pathway approaches are enduring 
and sustainable in at least 8 
colleges/state 

At least 8 colleges/state move = 
25% of target students to achieve 
marketable credentials & college 


credit without Dev Ed in < 4 years 
Robust, sustainable learning 
networks drive continuous 

__ program improvement 


ABE students supported (in 
academics, financial aid, 
college/career counseling) & 
integrated into colleges 
Students have knowledge of 
range of available supports 
Faculty, administrators & 
policymakers demonstrate 
commitment to ABE student 
success a 


States, colleges, employers & 
initiative partners understand 
costs & ROI 

3-4 viable financing models, 
including repurposed funding 
Participation at tipping point in 
multiple colleges (2 40% of target 
students) 

Sufficient data to inform 
replication 

First 4 states have changed 
policies to support scale beyond 
first 8 institutions 

Some federal policy barriers to 
financial aid & sustainable 
funding removed 


Long- Term Goals 


System 
Financially 
sustainable systems 
for integration of 
ABE students into 
marketable career 
pathways in college 
Consistent 
information flow on 
labor market 
demand & supply 
between business & 
colleges 
Colleges engage in 
continuous, data- 
driven program 
improvement 
Deep penetration of 
model in multiple 
colleges in diverse 
states 


Students 
High completion of 
multiple career 
pathway options for 
low-skill adults 
Skills, credit 
accumulation make 
college completion 
viable 
Improved job 
placement & 
employment results 
for participating 
students 


Notes: ABE = adult basic education; ASE = adult secondary education; CBOs = community-based organizations; ESL = English as a second language; ROI = return on investment; TA 


= technical assistance; WIBs = workforce investment boards. 
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Appendix B. AO Student 
Characteristics by State 


This appendix provides a detailed description of AO students in each state. 


AO Students in Illinois 


This section presents descriptive characteristics of AO students in Illinois, overall and in each year of 
implementation. AO students in Illinois were largely recruited from adult education, so students are not 
separated by recruitment source in these analyses, unlike in Kansas and Kentucky. The comparison 
group for Illinois is only drawn from non-AO adult education students. Statewide, there were 1.4 million 
adults without high school credentials and 550,000 English-language learners on average from 2009 to 


2011, constituting 11 percent and 4 percent of the total state population, respectively (OCTAE 2014). 


Selected Characteristics of the AO Population in Illinois 


Table B.1 presents some descriptive information about AO students in Illinois, divided into three 
occupational areas: health, manufacturing, and other (or occupations that were not clearly identifiable 


in the administrative data by course and credential histories). 


AO students in Illinois were relatively evenly divided between the three occupational areas defined 
for this analysis, with about 37 percent in health pathways, 33 percent in manufacturing pathways, and 
30 percent in other or unidentifiable pathways. Just over half of all students were female (55 percent), 
although the gender compositions of the students varied significantly by pathway. Over 92 percent of 
students in health pathways were female, while only about 9 percent of students in manufacturing 
pathways were female. The average age was around 30 for all AO students in Illinois, which was 
consistent with the other states. A plurality of students (almost 42 percent) were white, and there was a 
large share of black students as well (almost 39 percent). The racial and ethnic composition of AO 
students varied by pathway, with relatively fewer white and Hispanic students enrolled in health 
pathways and relatively more enrolling in manufacturing. Nearly one-third of students (30 percent) had 


ahigh school credential at entry. 
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TABLE B.1 
Selected Variables for Illinois AO Students, by Occupational Pathway 


AO students AO students 


AO students in in other or 
in health manufacturing unidentifiable 
AO students pathways pathways pathways 
Female (%) 55.4 92.2 9.2 60.6 
Average age at intake 28.9 30.0 28.7 27.8 
White (%) 41.6 34.8 50.4 40.2 
Black or African American (%) 38.9 52.0 27.5 35.2 
Hispanic or Latino (%) 16.7 11.3 21.5 18.2 
All other races and ethnicities (%) 1.8 1.9 NA 3.0 
Missing race or ethnicity (%) 1.0 0.0 NA 3.4 
High school diploma or HSE at entry (%) 29.8 32.6 35.2 20.5 
Greater than high school at entry (%) 0.0 0.0 0.0 0.0 
Pell grant recipient (%) 17.5 11.0 31.0 11.0 
Predicted probability of college (%) 35.9 41.5 30.2 35.1 
Average quarters employed in prior two years 2.9 3.2 3.2 2.3 
Sample size 867 319 284 264 


Notes: HSE = high school equivalency; NA = not applicable, values suppressed because of sample sizes of fewer than three 
observations. 


Overall, only 17.5 percent of AO students in Illinois received a Pell grant, although this figure was 
31 percent for students in manufacturing pathways. This may relate to the restriction on Pell grant 
eligibility for students who did not have high school credentials because the ability-to-benefit provision 
was terminated for the majority of AO implementation. The predicted probability of college for all AO 
students in Illinois was 35.9 percent, with higher probabilities for students in health pathways and 
lower probabilities for students in manufacturing pathways. Illinois AO students had less work 
experience in the two years prior to enrollment than AO students in the other three states, with an 


average of only 2.9 quarters of employment. 


CHANGES IN CHARACTERISTICS OVER TIME 

The characteristics of Illinois AO students over the course of implementation were more stable than 
other states. In the third year of the program, there were more black students (45 percent) and female 
students (61 percent) than in other program years, but none of this variation constituted a major trend 


in the composition of entering students over time. 
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FIGURE B.1 


Characteristics of Illinois AO Students over Time 


Data for new enrollees from college records 


Total enrollment, by calendar year 
Students 
350 


300 ——_— 


250 
200 


Gender, by calendar year 


=u Male Female 


55% 50% 


61% 


Year 1 Year 2 Year 3 


Educational attainment, by calendar year 


WHS diplomaor HSE ®Greater thanHS ®Nodiploma 


Year 1 Year 2 Year 3 


Age, by calendar year 


m1i7-19 m 20-22 © 23-26 
m27-35 36-54 m Over 54 
21% 23% 22% 


Year 1 Year 2 Year 3 


Race or ethnicity, by calendar year 


a White African American 


= Hispanic/Latino = Asian, multiple, other 


Year 1 Year 2 Year 3 


Recruitment source, by calendar year 


w Adult ed. Dev. ed. mCTE = Other 


Year 1 Year 2 Year 3 


Sources: Illinois Community College Board Adult Education and Postsecondary Records. 
Notes: CTE = career and technical education; HSE = high school equivalency; Dev. ed. = developmental education. Percentages 
are computed for students for whom data are available; missing values are excluded. Year 1 is 2012, year 2 is 2013, and year 3 is 


2014. 
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AO Students in Kansas 


Kansas relied heavily on the population of current CTE students for its recruitment; therefore, the 
impact analysis includes a second CTE comparison group in addition to the adult education comparison 
group. This section presents descriptive characteristics of Kansas AO students. Statewide, there were 
237,000 adults without high school credentials and 62,000 English-language learners on average from 
2009 to 2011, constituting 8 percent and 2 percent of the total state population, respectively (OCTAE 
2014). 


Selected Characteristics of the AO Population in Kansas 


Descriptive statistics for AO students are presented separately by recruitment source (adult education 


and CTE). 


CHARACTERISTICS OF ADULT EDUCATION STUDENTS 

Table B.2 presents basic information on AO students recruited from adult education. As was the case in 
Illinois, AO students in Kansas can be divided into three major occupational pathways: health 
occupations, manufacturing occupations, and other occupations or occupations that were not clearly 


identifiable in the administrative data by course and credential histories. 


Over 73 percent of AO students drawn from adult basic education programs were enrolled in 
health pathways. Seventy-seven percent of all adult education AO students and 89 percent of those in 
health pathways were female. Kansas adult education students were generally in their late twenties. 
Almost 43 percent of all Kansas adult education students were white, almost 20 percent were Hispanic 
or Latino, and just over 9 percent were black or African American. AO students in manufacturing 
pathways that were recruited from adult education were more likely to be black or African American 


and male than were students in other pathways. 


Surprisingly, Kansas AO students recruited from adult education had relatively high levels of 
educational attainment; 68 percent of that population had a high school diploma or HSE at entry, and 
over 7 percent had at least some college education before AO. These adult education students may 
have earned a high school credential shortly before AO enrollment and therefore were measured as 
having one at AO entry, even though nationally most adult basic education students do not have a high 
school credential. Nonhealth adult education students (largely men) had much lower levels of 
educational attainment than those in health pathways (mainly women). The substantial educational 


attainment of AO students recruited from adult education highlights the importance of matching them 
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to acomparison group of adult education students with similar characteristics when estimating the 


impact of AO. 


TABLE B.2 
Selected Variables for Kansas AO Students Recruited from Adult Education, by Occupational 
Pathway 
AO students AO students 
AO students in in other or 
in health manufacturing unidentifiable 
AO students pathways pathways pathways 
Female (%) 76.9 89.3 40.2 40.4 
Average age at intake 28.4 28.5 28.9 21.5 
White (%) 42.9 42.7 39.1 40 
Black or African American (%) 9.2 74 16.3 NA 
Hispanic or Latino (%) 19.8 20.8 14.1 60 
All other races and ethnicities (%) 4.8 4.5 6.5 NA 
Missing race or ethnicity (%) 23.3 24.6 23.9 NA 
HSE at entry (%) 16.8 15.4 22.8 NA 
High school diploma at entry (%) 51.4 53.7 42.4 80 
Greater than high school at entry (%) 7.6 9.5 NA NA 
Pell grant recipient (%) 24.6 23.7 27.2 40 
Predicted probability of college (%) 25.5 26.1 25.5 22 
Average quarters employed in prior two years 4.1 4.1 3.7 4.5 
Sample size 459 337 126 10 


Notes: HSE = high school equivalency; NA = not applicable, values suppressed because of sample sizes of fewer than three 
observations. 


Pell grant receipt among students recruited from adult education was low at just under 25 percent. 
The ability-to-benefit provision discussed above may account for this low Pell grant receipt rate. 
Generally, manufacturing students had a slightly lower probability of attending college than health 
students, though the differences are not large. Kansas AO students from adult education had recorded 


employment on average in only 4.1 of the eight quarters before AO enrollment. 


CHARACTERISTICS OF CTE STUDENTS 
Although adult education students were the original target population for AO, 73 percent of the 1,698 
AO students came from CTE. As is the case for the adult education students in Kansas, 58 percent of 


CTE students were in health pathways. Table B.3 presents descriptive statistics for the CTE population. 


TABLE B.3 
Selected Variables for Kansas AO Students Recruited from CTE before Match, by Occupational 
Pathway 


AO students AO students 


AO students in in other or 
in health manufacturing unidentifiable 
AO students pathways pathways pathways 
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AO students AO students 


AO students in in other or 
in health manufacturing unidentifiable 
AO students pathways pathways pathways 
Female (%) 52.9 86.0 6.5 11.4 
Average age at intake 27.3 27.3 27.0 30.6 
White (%) 54.9 58.9 50.9 31.8 
Black or African American (%) 16 12.6 17.9 52.2 
Hispanic or Latino (%) 15.9 14.3 18.7 11.4 
All other races and ethnicities (%) 5.8 6.0 5.7 * 
Missing race or ethnicity (%) 7.3 8.2 6.7 0.0 
HSE at entry (%) 9.5 7.1 12.8 11.4 
High school diploma at entry (%) 59.8 55.6 65.3 70.5 
Greater than high school at entry (%) 24.9 30.8 16.8 15.9 
Pell grant recipient (%) 43.5 33.3 56.8 68.2 
Average quarters employed in prior two years 4.1 4.5 3.8 2.6 
Sample size 1,239 719 475 44 


Notes: HSE = high school equivalency; NA = not applicable, values suppressed because of sample sizes of fewer than three 
observations. 


Out of all AO students recruited from CTE, almost 53 percent were female. The female share of 
students was higher in the health pathways (86 percent) and much lower in the manufacturing 
pathways (6.5 percent). The majority of students recruited from CTE (almost 55 percent) were white, a 
higher share than among the adult education population. The share of white students was high across all 
occupational pathways, but manufacturing and other pathways had a lower share of nonwhite students 
than health pathways. As may be expected, educational attainment for students from CTE was higher 
than it was for students from adult education, with over half of the population (59 percent) earning a 
high school diploma before entry, an additional 10 percent earning an HSE, and almost 25 percent had 
at least some college education. Pell grant receipt rates were nearly 44 percent among all CTE students; 
of students in manufacturing or other nonhealth pathways, well over half of all students (57 percent of 
manufacturing and 68 percent of students in other occupational areas) received Pell grants. Strikingly, 
the population recruited from CTE had approximately the same employment history as students 


recruited from adult education prior to AO enrollment (4.1 of the previous 8 semesters). 


The predicted probability of attending college is not estimated for students recruited from CTE, 
because all of these students were already enrolled in college before coming into contact with AO. Thus, 


there is no possibility that AO increased college access for this population. 


Changes in Characteristics over Time 


Over the course of implementation, the characteristics of Kansas AO students shifted slightly toward 


non-white male students. These trends are apparent in the mini-charts in figure B.2. 
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FIGURE B.2 
Characteristics of Kansas AO Students over Time 


Data for new enrollees from college records 


Total enrollment, by calendar year Age, by calendar year 

Students m17-19 @ 20-22 ™ 23-26 
800 m 27-35 36-54 mOver 54 
600 20% 18% 16% 


Year 1 Year 2 Year 3 Year 1 Year 2 Year 3 
Gender, by calendar year Race or ethnicity, by calendar year 
= Male Female g White African American 
© Hispanic/Latino = Asian, multiple, other 
14% 16% 21% 
63% 59% 57% 


15% 15% 


Year 1 Year 2 Year 3 Year 1 Year 2 Year 3 
Educational attainment, by calendar year Recruitment source, by calendar year 
@ HS diploma © HSE ® Greater than HS = No diploma m Adult ed. Dev.ed. =CTE mOther 
14% 


12% 12% 11% 


Year 1 Year 2 Year 3 Year 1 Year 2 Year 3 


Sources: Kansas Board of Regents Postsecondary Records. 
Notes: CTE = career and technical education; HSE = high school equivalency; Dev. ed. = developmental education. Percentages 
are computed for students for whom data are available; missing values are excluded. Year 1 is 2012, year 2 is 2013, and year 3 is 


2014. 
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AO Students in Kentucky 


Like Kansas, Kentucky recruited most of its AO participants from college developmental education or 
routed new students who would have needed to enroll in developmental education into AO. This 

section presents descriptive characteristics of Kentucky AO students. Statewide, there were 575,000 
adults without high school credentials and 39,000 English-language learners on average from 2009 to 


2011, constituting 13 percent and 1 percent of the total state population, respectively (OCTAE 2014). 


Selected Characteristics of the AO Population in Kentucky 


CHARACTERISTICS OF ADULT EDUCATION STUDENTS 

Table B.4 presents some basic information on AO students recruited from adult education. As was the 
case in Illinois and Kansas, AO students in Kentucky can be divided into three major occupational 
pathways: health occupations, manufacturing occupations, and other occupations or occupations that 


were not clearly identifiable in the administrative data by course and credential histories. 


TABLE B.4 

Selected Variables for Kentucky AO Students Recruited from Adult Education, by Occupational 
Pathway 

AO students AO students 
AO students in in other or 
AO in health manufacturing unidentifiable 
students pathways pathways pathways 

Female (%) 56.6 92.3 9.5 60.7 
Average age at intake 29.3 28.6 31.4 27.6 
White (%) 75.4 80.8 80.8 60.7 

Black or African American (%) 21.3 19.2 16.7 32.1 
Hispanic or Latino (%) 0.0 0.0 0.0 0.0 

All other races and ethnicities (%) 3.3 0.0 - i 
Missing race or ethnicity (%) 0.0 0.0 0.0 0.0 

HSE at entry (%) 14.8 15.4 16.7 10.7 

High school diploma at entry (%) 0.0 0.0 0.0 0.0 
Greater than high school at entry (%) 0.0 0.0 0.0 0.0 

Pell grant recipient (%) 32.8 32.7 21.4 50.0 
Predicted probability of college (%) 21.3 19.2 20.7 26.2 
Average quarters employed in prior two yearst 3.1 2.7 3.9 2.6 
Sample size 122 52 44 26 


Notes: HSE = high school equivalency; NA = not applicable, values suppressed because of sample sizes of fewer than three 
observations. 


Over 40 percent of AO students in Kentucky were enrolled in health pathways, a much lower figure 
than in Kansas but higher than in Illinois. This may relate to the characteristics of the adult education- 


recruited AO population. Fifty-seven percent of students from adult education were female, but over 
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92 percent of those in health pathways were female. Conversely, only about 10 percent of the students 
in manufacturing pathways were female. Kentucky adult education students were in their late twenties 
and early thirties, with somewhat younger students entering health pathways and somewhat older 
students entering manufacturing pathways. About 75 percent of all Kentucky adult education students 
were white, over 21 percent were Black or African American, and no AO students from adult education 
were Hispanic or Latino. The relatively few students in pathways other than health and manufacturing 
were somewhat more likely to be identified as Black or African American than students in health and 


manufacturing. 


About 15 percent of Kentucky students from adult education had an HSE at entry, which they 
would have earned in the semester immediately before AO enrollment, based on the definitional 
distinction between students recruited from adult education and those recruited from developmental 


education. None had a prior postsecondary credential. 


Pell grant receipt among students recruited from adult education was low - at 33 percent. Pell 
grant receipt was much higher for AO students in other (i.e., nonhealth, nonmanufacturing) pathways, at 
50 percent. Health students had the lowest probabilities of attending college, at about 19 percent, while 
the values for manufacturing and other students were somewhat higher. The average AO student 
recruited from adult education was employed for 3.1 of the 8 quarters preceding enrollment. Students 
in manufacturing pathways had higher prior employment rates than students in health and other 


pathways. 


CHARACTERISTICS OF DEVELOPMENTAL EDUCATION STUDENTS 

Most students in Kentucky were recruited from developmental education classes. AO students 
recruited from developmental education were relatively evenly split between health and manufacturing 
pathways with 55 percent in health, 38 percent in manufacturing, and the remainder in other pathways. 
Out of all AO students recruited from developmental education, almost 55 percent were female. As in 
other states, the female share of students was higher in health pathways (87 percent) and much lower in 
manufacturing pathways (6 percent). The large majority of students recruited from developmental 
education (84 percent) were white, which is about 8 percentage points higher than the share from adult 
education. The share of white students was high across all occupational pathways, but manufacturing 
and other pathways had a higher share of non-white students than health pathways. By construction, 
students from developmental education all had either a HSE or a high school diploma, and a small share 
had some postsecondary degree or certification. Pell grant receipt rates were about 50 percent among 
all developmental education students, with the highest rates among students in health pathways. The 
average AO student recruited from developmental education was employed for 3.5 of the eight 
quarters preceding enrollment. Unlike the adult education population, students in manufacturing and 


other pathways had somewhat lower prior employment rates. 
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Table B.5 presents descriptive statistics for the developmental education population. 


Out of all AO students recruited from developmental education, almost 55 percent were female. As 
in other states, the female share of students was higher in health pathways (87 percent) and much lower 
in manufacturing pathways (6 percent). The large majority of students recruited from developmental 
education (84 percent) were white, which is about 8 percentage points higher than the share from adult 
education. The share of white students was high across all occupational pathways, but manufacturing 
and other pathways had a higher share of non-white students than health pathways. By construction, 
students from developmental education all had either a HSE or a high school diploma, and a small share 
had some postsecondary degree or certification. Pell grant receipt rates were about 50 percent among 
all developmental education students, with the highest rates among students in health pathways. The 
average AO student recruited from developmental education was employed for 3.5 of the eight 
quarters preceding enrollment. Unlike the adult education population, students in manufacturing and 


other pathways had somewhat lower prior employment rates. 


TABLE B.5 
Selected Variables for Kentucky AO Students Recruited from Developmental Education, by 
Occupational Pathway 


AO students AO students 


AO students in in other or 
in health manufacturing unidentifiable 
AO students pathways pathways pathways 
Female (%) 54.8 86.9 5.7 72.2 
Average age at intake 29.5 28.6 30.8 28.6 
White (%) 83.9 85.4 83.3 75.6 
Black or African American (%) 10.8 8.6 12.7 16.7 
Hispanic or Latino (%) 2.9 2.8 2.5 5.6 
All other races and ethnicities (%) 2.4 3.1 1.5 NA 
Missing race or ethnicity (%) 0.0 0.0 0.0 0.0 
HSE at entry (%) 10.9 9.8 12.1 12.2 
High school diploma at entry (%) 89.1 87.5 87.1 86.7 
Greater than high school at entry (%) 1.8 2.7 0.9 NA 
Pell grant recipient (%) 49.7 54.0 43.6 48.9 
Average quarters employed in prior 2 yearst 3.5 3.8 3.0 2.9 
Sample size 1,234 674 472 88 


Notes: HSE = high school equivalency; NA = not applicable, values suppressed because of sample sizes of fewer than three 
observations. Columns do not sum to sample sizes because 14 AO participants had pathways that could not be identified 


CHANGES IN CHARACTERISTICS OVER TIME 
Over the course of implementation, the characteristics of Kentucky AO students shifted toward more 
male students. Changes over time in race, high school credential attainment, and recruitment source 


were not statistically significant at p < 0.10. The values are presented in the mini-charts in figure B.3. 
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FIGURE B.3 


Characteristics of Kentucky AO Students over Time 


Data for new enrollees from college records 
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Sources: Kentucky Community and Technical College System Postsecondary Records. 

Notes: CTE = career and technical education; HSE = high school equivalency; Dev. ed. = developmental education. Percentages 
are computed for students for whom data are available; missing values are excluded. Year 1 is 2012, year 2 is 2013, and year 3 is 
2014. 
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AO Students in Louisiana 


The characteristics of AO students in Louisiana differ markedly from the patterns in Kentucky and 
Kansas, largely because AO students in Louisiana were drawn only from the adult education system. AO 
students in Louisiana were more similar to AO students in Illinois; both states recruited primarily from 
adult education. Statewide, there were 601,000 adults without high school credentials and 50,000 
English language learners on average from 2009 to 2011, constituting 13 percent and 1 percent of the 
total state population, respectively (OCTAE 2014). It is important for the reader to remember that only 
AO students who took at least one credit-bearing class are included in these descriptive statistics, and 


these students constitute less than half of the students flagged as AO in the state data system. 


Selected Characteristics of the AO Population in Louisiana 


Table B.6 presents some descriptive information about Louisiana AO students overall and in the 
occupational areas of health, manufacturing, and other (or occupations that were not clearly 
identifiable in the administrative data by course and credential histories). The results in Table B.6 refer 
only to adult education students, given that Louisiana AO students were only drawn from one 


recruitment source. 


Similar to adult education students in Kentucky, AO were relatively evenly split between health and 
manufacturing pathways with 44 percent in health, 35 percent in manufacturing, and the remainder in 


other or unidentifiable pathways. 


TABLE B.6 
Selected Variables for Louisiana AO Students, by Occupational Pathway 


AO students 
AOstudents AOstudentsin in other or 
in health manufacturing unidentifiable 
AO students pathways pathways pathways 
Female (%) 64.1 90.1 33.1 61.7 
Average age at intake 25.8 26.5 24.0 27.6 
White (%) 36.8 39.1 38.3 30.9 
Black or African American (%) 50.2 50.0 49.3 50.6 
Hispanic or Latino (%) 2.7 2.6 3.3 NA 
All other races and ethnicities (%) 6.4 6.8 5.8 6.2 
Missing race or ethnicity (%) 3.9 1.6 3.3 9.9 
HSE at entry (%) 0.7 NA NA 0.0 
High school diploma at entry (%) 11.6 8.9 17.5 7.4 
Greater than high school at entry (%) 1.6 2.1 NA NA 
Pell grant recipient (%) 21.6 13.0 33.8 19.8 
Predicted probability of college (%) 9.5 10.0 9.6 7.6 
Average quarters employed in prior 2 years 3.5 3.7 3.3 3.2 
Sample size 440 192 154 81 


Notes: HSE = high school equivalency; NA = not applicable, values suppressed because of sample sizes of fewer than three 
observations. 
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Almost two thirds of AO students in Louisiana were female. As in other states, the female share of 
students was much higher in health pathways (over 90 percent), although not as low in manufacturing 
pathways as the other states (33 percent). About half of all AO students from Louisiana were black or 
African American, a share that held fairly consistently across occupational areas. Approximately one 
third of the total was white, with the remainder being Hispanic or Latino, being of another race or 
ethnicity or having missing information on race and ethnicity. AO students in Louisiana had low 
educational attainment levels, with 1 percent and 12 percent holding an HSE or a high school diploma at 
entry, respectively. Almost all the rest had not completed high school, although a small share (almost 2 
percent) had earned a postsecondary award of some kind. Pell grant receipt rates were 22 percent for 
AO students as a whole. This low rate of Pell grant receipt may be partially explained by the generous 
use of tuition waivers by Louisiana AO colleges, as permitted by the state. The predicted probability of 
attending college was low for AO students in all pathways, with a 9 percent probability of attending 
college on average. This indicates that the AO students recruited from adult education were not 
necessarily those who were most likely to be college-goers based on their test scores and 
demographics. The average AO student in Louisiana worked for 3.5 of the eight quarters prior to 
enrollment, with stronger employment histories for students in health pathways than in other 


occupational areas. 


CHANGES IN CHARACTERISTICS OVER TIME 
Over the course of implementation, the characteristics of Louisiana AO students shifted toward more 
male students (statistically significant at p < 0.10). There were no significant time trends in educational 


attainment or age. 
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FIGURE B.4 


Characteristics of Louisiana AO Students over Time 


Data for new enrollees from college records 
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Sources: Louisiana Community and Technical College System Postsecondary Records. 
Notes: CTE = career and technical education; HSE = high school equivalency; Dev. ed. = developmental education. Percentages 


are computed for students for whom data are available; missing values are excluded. Year 1 is 2012, year 2 is 2013, and year 3 is 
2014. 
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Appendix C. Labor Market Impact 
Results Tables 


This appendix reports detailed results for the employment and earnings impacts of AO, by state. 


TABLE C.1 


Employment and Earnings Impact Results for Illinois 


All comparison 


Outcomes by quarter after enrollment mean outcome All AO impact 
Employment (%-point) by quarter after enrollment 
Quarter 1 (total n = 4,996) 44.1% -2.9%** 
Quarter 2 (total n = 4,996) 46.7% -2.1%* 
Quarter 3 (total n = 4,996) 50.6% -1.1% 
Quarter 4 (total n = 4,996) 52.4% 3.3%*** 
Quarter 5 (total n = 4,981) 55.3% 4.9%*** 
Quarter 6 (total n = 4,968) 57.4% 4.2%*** 
Quarter 7 (total n = 4,473) 57.0% 2.5%* 
Quarter 8 (total n = 4,285) 58.0% 2.0% 
Quarter 9 (total n = 3,675) 59.9% 1.4% 
Quarter 10 (total n = 3,096) 58.6% 1.2% 
Quarter 11 (total n = 2,878) 55.8% 5.6%*** 
Quarter 12 (total n = 2,335) 53.5% 8.0%*** 
Earnings by quarter after enrollment 
Quarter 1 (total n = 4,996) $1,303 -$146** 
Quarter 2 (total n = 4,996) $1,376 -$184*** 
Quarter 3 (total n = 4,996) $1,682 -$63 
Quarter 4 (total n = 4,996) $1,815 $215*** 
Quarter 5 (total n = 4,981) $2,035 $293*** 
Quarter 6 (total n = 4,968) $2,468 $16 
Quarter 7 (total n = 4,473) $2,408 $67 
Quarter 8 (total n = 4,285) $2,533 $115 
Quarter 9 (total n = 3,675) $2,583 $156 
Quarter 10 (total n = 3,096) $2,494 -$52 
Quarter 11 (total n = 2,878) $2,718 -$62 
Quarter 12 (total n = 2,335) $2,799 -$26 


Note: Statistical significance is indicated as asterisks where *** p<0.01, ** p<0.05, * p<0.10. Impact estimates without 
coefficients are not statistically different from zero. 
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TABLE C.2 


Employment and Earnings Impact Results for Kansas 


By Recruitment Source 


Comparison 
All from adult Comparison 
comparison ed. AO from from CTE 
mean All AO mean Adult ed. mean AO from 
Outcome by quarter after enrollment outcome Impact outcome impact outcome CTE impact 
Employment (%-point) by quarter after enrollment 
Quarter 1 (total n= 11,652) 58.4% 3.7%*** 60.8% -0.4% 57.5% 4.7%*** 
Quarter 2 (total n = 10,479) 59.4% 5.0%*** 59.9% 2.1% 59.2% 5.1%*** 
Quarter 3 (total n = 9,336) 63.8% 3.8%*** 70.7% -1.3% 61.4% 5.3%*** 
Quarter 4 (total n = 9,336) 67.9% 1.5%* 75.3% -4.2% 65.5% 3.1%*** 
Quarter 5 (total n = 6,950) 66.2% 4.2%*** 69.8% 1.3% 64.8% 4.3%*** 
Quarter 6 (total n = 6,043) 68.7% 1.2% 78.5% -4.5% 65.0% 2.4%** 
Quarter 7 (total n = 4,716) 72.0% 1.3% 84.7% -8.3% 65.3% 6.9%*** 
Quarter 8 (total n = 4,716) 72.0% 0.2% 84.6% -8.3%* 65.4% 5.5%*** 
Earnings by quarter after enrollment 
Quarter 1 (total n= 11,652) $2,142 $233*** $1,995 $46 $2,198 $325*** 
Quarter 2 (total n = 10,479) $2,468 $507*** $2,300 $346* $2,527 $565*** 
Quarter 3 (total n = 9,336) $2,598 $490*** $2,907 $118 $2,495 $676*** 
Quarter 4 (total n = 9,336) $3,010 $553*** $3,173 -$20 $2,955 $734*** 
Quarter 5 (total n = 6,950) $3,251 $466*** $3,411 $63 $3,188 $605*** 
Quarter 6 (total n = 6,043) $3,896 $350*** $4,610 -$284 $3,630 $610*** 
Quarter 7 (total n = 4,716) $3,551 $630*** $3,832 -$150 $3,402 $964*** 
Quarter 8 (total n = 4,716) $3,666 $811*** $3,779 $53 $3,606 $1,188*** 
Note: Statistical significance is indicated as asterisks where *** p<0.01, ** p<0.05, * p<0.10. 
TABLE C.3 
Employment and Earnings Impact Results for Kentucky 
By Recruitment Source 
Comparison 
All from adult Comparison 
comparison ed. AO from from dev. AO from 
mean All AO mean adult ed. ed. mean dev. ed. 

Outcome by quarter after enrollment outcome impact outcome impact outcome impact 
Employment (%-point) by quarter post enrollment 
Quarter 1 (total n = 20,011) 42.7% -0.3% 42.3% -1.2% 42.8% -0.8% 
Quarter 2 (total n = 20,011) 43.4% 1.7%*** 46.0% -5.7%*** 43.2% 2.4%*** 
Quarter 3(total n = 20,011) 46.5% 4.6%*** 47.8% -1.1% 46.3% 4.9%*** 
Quarter 4 (total n = 20,011) 48.0% 3.0%*** 42.6% 4.8%*** 48.5% 2.4%*** 
Quarter 5 (total n = 20,011) 50.3% 2.8%*** 46.9% 6.4%*** 50.6% 1.7%** 
Quarter 6 (total n = 20,011) 49.0% 3.5%*** 45.0% 6.0%*** 49.4% 3.1%*** 
Quarter 7 (total n = 20,011) 52.0% 2.3%*** 46.7% 10.0%*** 52.6% 1.4%** 
Quarter 8 (total n = 17,134) 50.6% 1.5%** 45.5% 2.9% 51.1% 1.7%** 
Quarter 9 (total n = 16,503) 52.9% 3.2%*** 47.0% 4.6%** 53.4% 2.5%*** 
Quarter 10 (total n= 14,129) 53.6% 2.7%*** 50.6% 2.8% 53.8% 1.9%** 
Quarter 11 (total n= 14,129) 55.8% 0.7% 46.3% 3.2% 56.7% -0.4% 
Quarter 12 (total n = 10,630) 54.3% 2.5%*** 49.5% 1.9% 54.9% 3.6%*** 
Earnings by quarter post enrollment 
Quarter 1 (total n = 20,011) $1,687 -$198*** $2,145 -$701*** $1,642 -$221 
Quarter 2 (total n = 20,011) $1,703 -$132*** $2,164 -$913*** $1,657 -$16 
Quarter 3(total n = 20,011) $1,922 $23 $2,162 -$133 $1,898 $74* 
Quarter 4 (total n = 20,011) $2,114 -$22 $1,854 $342*** $2,140 -$59 
Quarter 5 (total n = 20,011) $2,383 -$43 $2,085 $575*** $2,412 -$79* 
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By Recruitment Source 
Comparison 
All from adult Comparison 
comparison ed. AO from from dev. AO from 
mean All AO mean adult ed. ed. mean dev. ed. 

Outcome by quarter after enrollment outcome impact outcome impact outcome impact 
Quarter 6 (total n = 20,011) $2,344 -$129*** $2,172 $156 $2,361 -$99** 
Quarter 7 (total n = 20,011) $2,583 -$92* $2,247 $505*** $2,616 -$86* 
Quarter 8 (total n = 17,134) $2,522 -$108** $1,826 $436*** $2,589 -$71 
Quarter 9 (total n = 16,503) $2,915 -$155*** $1,984 $855*** $3,004 -$163*** 
Quarter 10 (total n= 14,129) $2,652 -$87 $2,161 $827*** $2,699 -$12 
Quarter 11 (total n= 14,129) $2,885 $49 $2,233 $629*** $2,948 $104 
Quarter 12 (total n= 10,630) $2,952 $111 $2,118 $687*** $3,045 $249*** 


Note: Statistical significance is indicated as asterisks where *** p<0.01, ** p<0.05, * p<0.10. 
TABLE C.4 
Employment and Earnings Impact Results for Louisiana 


Outcomes by quarter after enrollment Ameemparean 


mean outcome All AO impact 
Employment (%-point) by quarter post enrollment 
Quarter 1 (total n = 3,455) 48.6% 6.3%*** 
Quarter 2 (total n = 3,455) 51.9% 6.2%*** 
Quarter 3 (total n = 3,455) 53.5% 4.4%*** 
Quarter 4 (total n = 3,209) 52.1% 7.3%*** 
Quarter 5 (total n = 2,751) 50.2% 10.8% *** 
Quarter 6 (total n = 2,751) 49.1% 9.1%*** 
Quarter 7 (total n = 2,458) 56.7% -1.8% 
Quarter 8 (total n = 2,440) 57.8% -4.8%** 
Quarter 9 (total n = 2,418) 60.5% -3.9%** 
Quarter 10 (total n = 2,418) 62.6% 0.7% 
Quarter 11 (total n= 1,549) 51.4% 5.4%** 
Quarter 12 (total n = 1,302) 62.2% -1.0% 
Earnings by quarter post enrollment 
Quarter 1 (total n = 3,455) $1,532 $205*** 
Quarter 2 (total n = 3,455) $1,721 $166** 
Quarter 3 (total n = 3,455) $1,884 $193** 
Quarter 4 (total n = 3,209) $1,988 $316*** 
Quarter 5 (total n = 2,751) $1,868 $710*** 
Quarter 6 (total n = 2,751) $1,920 $500*** 
Quarter 7 (total n = 2,458) $2,231 -$212** 
Quarter 8 (total n = 2,440) $2,564 -$363*** 
Quarter 9 (total n = 2,418) $2,983 -$500*** 
Quarter 10 (total n = 2,418) $3,278 -$610*** 
Quarter 11 (total n = 1,549) $3,041 -$491*** 
Quarter 12 (total n = 1,302) $3,686 -$633*** 


Note: Statistical significance is indicated as asterisks where *** p<0.01, ** p<0.05, * p<0.10. 
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Appendix D. Balancing Tests 


This appendix reports output from the Stata “pstest” command, which produces balancing tests. These 
represent balancing tests for the match of the entire sample, as utilized for reporting education impacts. 
Earnings and employment impacts were based on rematched samples to account for the changing 
sample composition caused by right-hand time censoring of later cohorts. The results of balancing tests 


for the quarterly samples are available from the authors upon request. 


These results show the balance of the treatment and comparison samples before regression 
adjustment. The regression adjustment should control for remaining imbalances in observable 
characteristics. Some characteristics may remain unbalanced after the propensity score matching 
exercise because the large number of matching variables relative to the sample makes it difficult to 


exactly align each matching variable. 


Illinois 


TABLE D.1 


Balancing Test Results for Propensity Score Match in Illinois, All Students 


Unmatched Mean Sreduct t-test V(T)/ 
Variable Matched Treated Control Sbias |bias| c o> | V(C) 
abeged_collegeintent U -3586 -44285 -44.1 -12.18 0.000 1.20* 
M 3586 37372 =7.9 82.1 #1,59 0,111 1.04 
enroll term201201 U .09689 .33907 =61.3 -14.48 0.000 
.09689 12118 -6.2 90.0 -1.62 0.105 
enroll term201202 U .05075 .05619 -2.4 -0.64 0.524 
.05075 .05475 -1.8 26.4 -0.37 0.710 
enroll _term201203 U 18454 12715 15.9 4.47 0.000 
M 18454 8511 -0.2 99.0 -0.03 0.976 
enroll _term201301 U .09112 . 10366 -4.2 =-L.1l 0.266 
-O09112 .08462 262 48.2 0.48 0.633 
enroll _term201303 U 1857 10729 22.3 6.46 0.000 
M 1857 8322 .7 96.8 0.13 0.894 
enroll term201401 U 16609 11213 15.6 4.43 0.000 
16609 13245 Oe 37.6 1.97 0.049 
enroll term201402 U 03114 03221 -0.6 -0.16 0.871 
M 03114 036 -2.8 -354.5 -0.56 0.575 
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Kansas 


TABLE D.2 


Balancing Test Results for Propensity Score Match in Kansas, All Students 
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Unmatched 
Variable Matched 
collegeintent U 
M 

semesterl U 
M 

semester2 U 
M 

semester3 U 
M 

semester4 U 
M 

semester5 U 
M 

semester6 U 
M 

semester? U 
M 

semesters U 
M 

semester9 U 
M 

NRS low U 
M 

NRS3 U 
M 

NRS4 U 
M 

NRS5 U 
M 

NRS6 U 
M 

NRSmissing U 
M 

priorged U 
M 

highschool U 
M 

gthighschool U 
M 
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20259 


Mean 
ad Control 


Sbias 


Sreduct 
|bias | 


86.3 


71.2 


42.9 


92.9 


95.8 


87.2 


84.9 


98.7 


=37.7 


76.3 


Pls 


98.2 


t-test 

t p>|tl 
31.54 0.000 
-0.82 0.410 
-7.70 0.000 
=1513.- 0.254 
-5.91 0.000 
-1.57 0.116 
-1.91 0.056 

0.87 0.383 


0.53 0.593 
4.06 0.000 
-0.11 0.912 
-2.12 0.034 
-0.21 0.831 


1.22 0.221 
-3.99 0.000 
-0.05 0.962 
-6.05 0.000 
-0.73 0.466 

0.36 0.717 

0.36 0.719 

8.37 0.000 

0.80 0.425 


-0.00 1.000 
4.47 0.000 
0.35 0.730 


45.63 0.000 


-0.25 0.802 


singleparent U .02532 -04367 -10.1 -3.56 0.000 

M = 02532 02607 -0.4 95:.9 -0.14 0.891 
age U 27.572 25.851 18.0 7.35 0.000 1.333% 

M 27.572 27.847 22:29 84.0 -0.79 0.431 1.03 
agesq U 864.9 747.27 LPB 7.40 0.000 1.42* 

M 864.9 877.55 =1.9 89.2 -0.52 0.603 1.04 
female U - 59364 -55284 8.3 3.18 0.001 

M - 59364 - 57008 4.8 42.3 1.39 0.164 
white U -51708 68035 =33:'8 -13.42 0.000 

M -51708 49544 4.5 86.7 1.26 0.207 
black U .14193 - 08408 18.3 7.80 0.000 

M 14193 14011 0.6 96.8 0.15 0.879 
hispanic U - 16961 - 10965 17.4 7.24 0.000 

M 16961 -21102 =12.0 30.9 -3.08 0.002 
otherrace U 05536 05296 esi 0.41 0.679 

M 05536 05117 1.8 -74.3 0.54 0.587 
missingrace U ~LL602 ~UI297 14.8 6.22 0.000 

M 11602 10225 4.7 68.0 1.29 0.198 
pell U 38457 - 32926 11.6 4.53 0.000 

M 38457 -36987 Sieclk 73.4 0.88 0.377 
collegeintent U 06906 00987 56.0 31.54 0.000 5.93% 

M 06906 07299 =3 5) 93.4 -0.82 0.410 0.96 
collegeintentmissing U ~f2965 - 96356 -68.6 =38.79 0.000 

M - 72968 - 72968 -0.0 100.0 -0.00 1.000 
semesterl U .03534 09035 -22.8 -7.70 0.000 

M .03534 . 04288 231 86.3 =1<13. 10.257 
semester2 U 05124 09488 -16.8 -5.91 0.000 

M 05124 06379 -4.8 Ts2 -1.57 0.116 
semester3 U -14075 15871 -5.0 -1.91 0.056 

M -14075 13049 2.29 42.9 0.87 0.383 
semester4 U .17903 .0813 29.3 13.10 0.000 

M 17903 17205 Aoi. 92.9 0.53 - 0.593 
semester5 U - 08598 06042 9.8 4.06 0.000 

M 08598 -08705 -0.4 95.8 -0.11 0.912 
semester6 U 149 16943 =§:.:6 -2.12 0.034 

M 149 -15162 -0.7 87.2 -0.21 0.831 
semester7 U .1702 06749 3201 14.78 0.000 

M 1702 15472 4.8 84.9 1.22 (0.221 
semester8 U -05713 08543 -11.0 =3.99 0.000 

M 05713 05751 -0.1 98.7 -0.05 0.962 
semester9 U 23133. 19198 =16.5 -6.05 0.000 

M .13133 .13989 -2.3 85.9 -0.73 0.466 
ACT U -.03924 -.0128 zr eylleesp! 8 -4.43 0.000 Lees 

M -.03924 -.04602 2.8 74.3 0.78 0.437 0.91 
ACTmissing U 92815 -91909 3.4 1.29 0.196 
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ACTsq 


COMPASS 


COMPASSmissing 


COMPASSsq 


ACCUPLACER 


ACCUPLACERmissing 


ACCUPLACERSq 


priorged 


highschool 


gthighschool 


singleparent 


age 


agesq 


female 


white 


black 


hispanic 


otherrace 


missingrace 


pell 


health 


manufacturing 
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92815 


06322 
06322 


03475 
03475 


- 89399 
- 89399 


-09953 
09953 


00236 
00236 


-9629 
-9629 


02591 
02591 


11484 
-11484 


57538 
-57538 


-20259 
20259 


-02532 
02532 


27.572 
27.572 


864.9 
864.9 


-59364 
- 99364 


-51708 
-51708 


14193 
14193 


-16961 
-16961 


-05536 
05536 


11602 
11602 


38457 
38457 


-62191 
-62191 


~33392 
- 33392 


- 93581 


05233 
06988 


.00024 
.06144 


93545 
- 86082 


-05804 
13915 


00135 
00087 


-99111 
- 96983 


.01024 
.02194 


.08241 
- 11108 


8447 
-56694 


00889 
-20607 


-04367 
-02607 


25.851 
27.847 


747.27 
8771355 


55284 
-57008 


- 68035 
-49544 


-08408 
14011 


10965 
21102 


-05296 
05117 


.07297 
10225 


-32926 
36987 


-66217 
.59714 


2663 
3432 


62. 


14. 
=2. 


15:5 


38. 


226 


20. 


39 


75. 


74. 


88. 


96. 


98. 


95. 


84. 


89. 


42. 


86. 


96. 


30. 


-74, 


68. 


73° 


38. 


86. 


376 


-167 
2592 


.000 
«023 


000 
003 


000 
054 


193 
- 780 


-000 
263 


.003 
O72 


.000 
- 730 


000 
-619 


.000 
-802 


.000 
891 


-000 
-431 


.000 
-603 


-001 
-164 


000 
w20F 


.000 
879 


000 
002 


-679 
-587 


-000 
.198 


000 
a oTT 


001 
£139 


.000 
-568 


~24* 
74% 


=69* 
+ 13% 


-40* 
5o* 


.53* 
-Le* 


<38* 
~18* 


othercip U - 41932 -56776 -30.0 -11.60 0.000 
M 41932 - 41536 0.8 97.3 0.23 0.815 
emptotal U 4.1243 4.6314 =16.3 -6.25 0.000 0.96 
M 4.1243 4.1441 -0.6 96.1 -0.18 0.854 0.95 
emp1 U -55418 - 64264 =18..1 =Tell 0.000 
M -55418 97192 =3.6 19.29) -1.04 0.298 
emp2 U -55418 60889 =11.1 -4.33 0.000 
M -55418 -54416 2.0 81.7 0.59 0.558 
earnl U 2244.1 2763.2 =13..9 -5.34 0.000 0.971 
M 2244.1 2244.1 0.0 100.0 0.00 1.000 w2il* 
earn2 U 2289.3 2654.1 =9..1 -3.67 0.000 0.87* 
M 2289.3 2324.6 =05.9 90.3 -0.29 0.776 O01 
earn3 4 U 4675.4 D2 1.2 -8.2 -3.12 0.002 0.92 
M 4675.4 4742.3 =0..:9 88.8 -0.28 0.782 -05 
earns 8 U 8465.6 9405.3 -6.7 =2.95 0.011 0.96 
M 8465.6 8851.7 =2 61 98.29 -0.81 0.418 -02 
unemployment U 6.1962 5.5941 50.2 18.17 0.000 0.69* 
M 6.1962 6.0694 10.6 78.9 3.30 0.001 0.88* 


* if variance ratio outside [0.91; 1.10] for U and [0.91; 1.10] for M 


Sample | Ps R2 LR chi2 p>chi2 MeanBias MedBias B R SVar 
—_—_—- 4--------------------- ~~ eee eee eee 
Unmatched | 0.291 3033.55 0.000 20.8 14.9 146.2* 2.57% 78 
Matched | 0.015 72:<92 0.007 3.2 23 29.5% 0.97 39 


* if B>25%, R outside [0.5; 2] 


TABLE D.3 


Balancing Test Results for Propensity Score Match in Kansas, Adult Education Students 


Unmatched Mean Sreduct t-test V(T)/ 
Variable Matched Treated Control Sbias |bias| © p>ltl V(C) 
collegeintent U -25547 -27087 =10.2 -1.54 0.124 0.96 
M 25547 27002 -9.6 565 -1.52 0.130 1.13 
semesterl U - 05882 -08715 -10.9 -1.65 0.099 
M .05882 .06116 -0.9 91.8 -0.15 0.882 
semester2 U 11765 .07843 3.2 2.00 0.046 
M 765 12909 -3.9 70.8 -0.53 0.599 
semester3 U .11547 ~L3725 =6.56 =“U,.99 0.321 
M 547 .07419 2.4 -89.5 2.14 0.033 
semester4 U -11547 -11765 =0.7 =0,10 0.918 
M 11547 .13903 -7.3  -981.4 -1.07 0.285 
semester5 U Pasi gi gig 09368 57 0.87 0.384 
M ell .14465 il sl re -92,4 -1.52 0.128 
semester6 U -07625 - 16776 =28:.2 -4.27 0.000 
M -07625 -09908 =7.:0 75.1 =1.22 0.222 
semester7 U . 18083 11547 1835 2.80 0.005 
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semester8 


semester9 


NRS low 


NRS3 


NRS4 


NRS5S 


NRS6 


NRSmissing 


priorged 


highschool 


gthighschool 


singleparent 


age 


agesq 


female 


white 


black 


hispanic 


otherrace 


missingrace 


pell 


health 
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=o 


33. 


96. 


66. 


88. 


-40. 


-116. 


Ee pees 


54. 


98. 


81. 


49. 


89. 


39. 


48. 


91. 5 


86. 


35% 


92 


74. 


62. 


75. 


-0. 


10. 


-000 
- 837 


.000 
074 


-001 
181 


075 
-820 


.008 
s122 


-O11 
+213 


-000 
410 


.000 
341 


-108 
23595 


028 
042 


314 
- 938 


-000 
-116 


-001 
wis 


.000 
086 


e23* 
-O1 


.30* 
-00 


manufacturing U -20044 39651 -43.8 -6.64 0.000 
M -20044 225275: -11.7 7323) -1.89 0.058 
othercip U 41394 - 61002 -40.0 -6.05 0.000 
M -41394 . 32394 18.3 54.1 2.84 0.005 
emptotal U 4.0697 4.0501 0.6 0.10 0.924 1.05 
M 4.0697 4.1985 =4.2 -556.6 -0.63 0.526 1.07 
emp1 U -91416 55991 -9.2 -1,39 0.165 
M -51416 -57178 -11.6 -25.9 -1.75 0.080 
emp2 U 2O2723 54031 =2:26 -0.40 0.692 
M «92723 53676 =1.9 271. =0.29 0.773 
earnl U 1866.2 2378.4 -14.0 -2.12 0.034 0..55* 
M 1866.2 1928.9 =1.7 87.8 -0.31 0.759 0.98 
earn2 U 1972.4 2261.8 -7.8 -1.18 0.238 0.51* 
M 1972.4 2124.1 -4.1 47.6 -0.70 0.485 0.75* 
earn3 4 U 4115.4 4890.9 =10.1 =1.53 0.127 0:..3.9* 
M 4115.4 4502.2 =5.0 50.1 -0.96 0.336 0.82* 
earns 8 U 7828.6 9579.6 =11.1.3.0 -1.67 0.096 0.38* 
M 7828.6 8861.4 -6.5 41.0 -1.29 0.199 0.89 
unemployment U 6.0473 5.8593 16.9 2.57 0.010 1.00 
M 6.0473 5.9833 5.8 66.0 0.86 0.391 0.93 


* if variance ratio outside [0.83; 1.20] for U and [0.83; 1.20] for M 


Sample | Ps R2 LR chi2 p>chi2 MeanBias MedBias B R SVar 
——— ee ee 4--------------- - - ee eee ee 
Unmatched | 0.342 435.02 0.000 20.4 11.8 159.6* 0.98 67 
Matched | 0.049 62.37 0.004 7.4 6.9 53:01k* 1.38 22 


* if B>25%, R outside [0.5; 2] 


TABLE D.4 
Balancing Test Results for Propensity Score Match in Kansas, CTE Students 


Unmatched Mean Sreduct t=test V(T)/ 

Variable Matched Treated Control Sbias |bias| t p>ltl V(C) 
semesterl U -02663 -09047 -27.4 -7.71 0.000 

M 02663 03611 -4.1 85.2 =1.35. 0.176 
semester2 U 02663 .0955 =29..1 -8.12 0.000 

M 02663 0396 =5.5 81.2 -1.80 0.071 
semester3 U 15012 ,L5953. =2:5.6 -0.86 0.388 

M 15012 , 15135. -0.3 86.9 -0.09 0.932 
semester4 U - 20258 -07993 35.8 14.39 0.000 

M -20258 .18428 53 85.01 1.15 0.249 
semester5 U .07667 .05916 7.0 2.46 0.014 

M .07667 .06572 4.4 37.4 1.06 0.289 
semester6 U 17595 1695 1g 0.58 0.565 

M ~17595 -17109 13.3 24.7 0.32 0.750 
semester7 U 16626 .06567 31.8 12.88 0.000 
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semester8 


semester9 


ACT 


ACTmissing 


ACTsq 


COMPASS 


COMPASSmissing 


COMPASSsq 


ACCUPLACER 


ACCUPLACERmissing 


ACCUPLACERSq 


priorged 


highschool 


gthighschool 


singleparent 


age 


agesq 


female 
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black 
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otherrace 


APPENDIX D 


16626 


- 04036 
04036 


13479 
13479 


-04342 
-04342 


- 92333 
- 92333 


06964 
-06964 


-04278 
-04278 


-88781 
88781 


10896 
- 10896 


-00161 
-00161 


954 
954 


- 03228 
- 03228 


-09524 
-09524 


59806 
-59806 


- 24939 
- 24939 


-02018 
02018 


27.282 
27.282 


847.13 
847.13 


52865 
-52865 


-54964 
-54964 


-16061 
-16061 


wAL9 
159 


05811 
-05811 


-16121 


-08644 
.04143 


.1938 
14922 


-.01307 
-.0474 


- 91636 
92264 


05388 
.07042 


- 00066 
-.04766 


- 93878 
-87765 


05686 
»12672 


-00148 
-.00057 


99102 
-95971 


-01038 
-02902 


- 06304 
-08819 


86898 
- 60832 


-00824 
~24484 


-04285 
02226 


25.5822 
27.265 


745.4 
842.18 


295521 
-50501 


- 68095 
-53146 


-08487 
1513 


10836 
19498 


05257 
-05278 


slo 
les 


18). 
3 


10. 


=13%5 
=1. 


95. 


OTs 


1D. 


86. 


90. 


95:. 


88. 


80. 


65. 


66. 


84. 


85. 


78. 


96. 


98. 


90. 


98. 


95. 


1a 


86. 


87. 


28. 


734 


-000 
-892 


.000 
-304 


000 
- 704 


-398 
-949 


.089 
-958 


-000 
»721 


-000 
432 


-000 
-470 


351 
882 


.000 
484 


.000 
~729 


-000 
-544 


.000 
602 


-000 
IS 


000 
.719 


.000 
-965 


-000 
-860 


073 
239 


.000 
-364 


000 
2923 


-000 
.019 


407 
562 


-26* 


oe 


Pas Se 
.04 


-41* 
07 


missingrace U -07264 07325 -0.2 -0.08 0.937 
M .07264 -06948 1.2 -415.0 0.31 0.760 
pell U -43584 32877 22.2 7.60 0.000 
M 43584 -40226 6.9 68.6 1.69 0.090 
health U -58031 -66727 -18.0 -6.16 0.000 
M -58031 56548 Se 83.0 0.75 0.456 
manufacturing U - 38337 26137 26.3 9.21 0.000 
M - 38337 3767 1.4 94.5 0.34 0.732 
othercip U 42131 -56617 =29.3 -9.80 0.000 
M 42131 -44923 =5.6 80.7 -1.40 0.161 
emptotal U 4.1445 4.6534 -16.4 -5.43 0.000 0.95 
M 4.1445 4.1239 0.7 96.0 0.16 0.871 0.90 
emp1 U 56901 - 64576 -15.8 -5.36 0.000 
M 56901 -57197 -0.6 96.1 -0.15 0.882 
emp2 U -56416 61149 =9.6 -3.25 0.001 
M 56416 -54691 3.5 63.5 0.86 0.388 
earnl U 2384.1 2777.8 =1023 -3.50 0.000 -08 
M 2384.1 2360.9 0.6 94.1 0.16 0.875 -28* 
earn2 U 2406.7 2669 -6.8 -2.28 0.022 0.97 
M 2406.7 2398.9 0.2 91s i 0.05 0.959 -09 
earn3 4 U 4882.8 5285.5 -5.4 -1.83 0.067 06 
M 4882.8 4831.3 0.7 87.2 0.17 0.861 ei. 
earns 8 U 8701.6 9398.7 -4.9 -1.66 0.098 -09 
M 8701.6 8848.1 =1..'0 79.0 -0.25 0.801 -05 
unemployment U 6.2513 5.5841 55.8 17.33 0.000 0.66* 
M 6.2513 6.1014 12:5 The 3.35 0.001 0..85* 


* if variance ratio outside [0.89; 1.12] for U and [0.89; 1.12] for ™M 


Sample | Ps R2 LR chi2 p>chi2 MeanBias MedBias B R SVar 
——— ee wa He ~~~ - ~~ ~~ ~~ - ~~ He ~~ eH 
Unmatched | 0.255 2108.14 0.000 18.1 15:52 130.1% 3 107% 67 
Matched | 0.012 Al 231 0.413 2.7 27 2529% 0.92 40 


* if B>25%, R outside [0.5; 2] 


Kentucky 


TABLE D.5 
Balancing Test Results for Propensity Score Match in Kentucky, All Students 


Unmatched | Mean Sreduct | t-test | v(T)/ 
Variable Matched | Treated Control Sbias |bias| | tc p>ltl | Vv(c) 
-------------------------- $----------------------------- == - =f 5-5-5 55 5 5 5-5 $5 5 === 
unemployment UU | 9.2107 7.9434 40.9 | 28.70 0.000 | 2.67* 
M | 9.1934 8.9381 8.2 19:29:.\| 1.89: 0059 | 1.28% 
| | | 
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collegeintent 


collegeintentmiss 


preAOdeg 


preAOPS 


preAOAE 


ageEnroll 


ageEnroll2 


ageEnrollMiss 


female 


race black 


race _otherunknown 


singleparentAO 


singleparentAOmiss 


pellAo 


aoEnrollSem201202 


aoEnrollSem201203 


aoEnrollSem201301 


aoEnrollSem201302 


aoEnrollSem201303 


aoEnrollSem201401 


aoEnrollSem201402 


aoEnrollSem201403 


CIPhealth 
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12081 
-12109 


-00441 
00369 


01618 
01622 


-51103 
51254 


-2875 
28835 


29.453 
29.443 


990.17 
989.6 


-26. 


58. 


=118. 


89. 


87. 


96. 


5% 


S69. 


99... 


50. 


94. 


90. 


99. 


=21.5 


74. 


47. 


88. 


=111. 


60. 


82. 


81. 


91. 


000 
000 


421 
764 


434 
e267 


-000 
-440 


.000 
O11 


-001 
928 


-001 
-916 


2393: 
290 


-000 
-990 


-389 
744 


002 
-884 


000 
«196 


.000 
#993 


so22 
2595 


.000 
.165 


-065 
-449 


005 
~752 


422 
e234. 


.000 
040 


-005 
-631 


.000 
e299 


-000 
.156 


0.29* 
0.26* 


CIPmanu U .39118 - 10432 70.4 31.78 0.000 
M 39012 -41813 -6.9 90.2 -1.49 0.137 
CIPanyother U . 66176 .90968 -63.4 -29.13 0.000 
M 66298 .67279 =25 96.0 -0.54 0.588 
emptotal U 3.4949 3.6059 -3.6 -1.26 0.207 0.98 
M 3.4941 3.3891 3.4 535 0.88 0.380 1.00 
earningsl U 1703.9 1985.8 -8.1 -2.80 0.005 0.87* 
M 1709 L715:..9 =0.2 OPS -0.05 0.958 0.93 
earnings2 U 1860 1:9°61..3 =2°7 =1.01 “0.392 12% 
M 1859.9 1754.8 2.8 -3.8 0.73 0.465 1517 
earnings3 4 U 4086.4 3913.8 2.3. 0.83 0.409 0.95 
M 4065.3 3917.5 2.0 14.4 0.53 0.596 1.01 
earnings5 8 U 8124.4 7082.9 7.0 2.72 0.007 1.41* 
M 8103.6 7730.3 25 64.2 0.63 0.528 1.6% 
emp1 U -45441 -49209 6 -2.68 0.007 
M “45575 - 44374 2.4 68.1 0.63 0.530 
emp2 U - 44632 -47778 =6.3 -2.24 0.025 
M -44617 4337 2.5 60.4 0.65 0.514 
earningslsq U 1.4e+07 1.7e+07 =3.6 -1.15 0.249 0.53* 
M 1.4e+07 1.5e+07 S11 69.0 -0.33 0.742 0.76* 
earnings2sq U 1.9e+07 1.6e+07 Bed L.2l 0.225 3.46* 
M 1.9e+07 1.6e+07 2.8 -14.8 0.73 0.468 3.06* 
earnings3 4sq U 7.0e+07 7.1le+07 =O... -0.04 0.972 0.04* 
M 6.9e+07 6.8e+07 0.2 -34.0 0.10 0.918 0.21* 
earnings5 8sq U 3.2e+08 2.3e+08 3.5 1.30 0.192 1.21% 
M 3.2e+08 2.8e+08 1.6 S37 0.51 0.609 4.03* 
NRS3 U .01985 0210 -0.8 -0.29 0.773 
M -01844 02273 23.0: $270.1 -0.79 0.431 
NRS4 U - 03676 -0342 1.4 0.50 0.616 
M -0354 20312 2.3. -63.1 0.61 0.544 
NRS5 U 01103 0126 =1..5 -0.50 0.615 
M 01106 00803 2.8 =93.62 0.81 0.418 
NRS6 U - 00588 -01421 -8.4 -2.56 0.011 
M 0059 -00573 0.2 98.0 0.06 0.954 
NRSm U s0375 - 16071 -42.1 =12.25 0.000 
M -03761 - 18853 -51.6 =22..5 -12.77 0.000 
ACTscoreStd U -.01017 -.01919 3.0 1.11  O.267 1.24* 
M -.0102 -.01003 =Oe1 98.1 -0.01 0.989 1.08 
ACTscoreStd2 U - 10233 .08271 4.0 1.46 0.145 1.16* 
M . 10264 0946 1.6 59.1 0.40 0.688 0.95 
ACTscoreStdMissing U - 78676 -7569 valk 2.49 0.013 
M 78909 . 79669 -1.8 74.5 -0.49 0.625 
* Lf variance ratio outside [0.90; 1.11] for U and [0.907 1.11] for M 
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Unmatched 
Matched 


Ps R2 LR chi2 p>chi2 MeanBias 


89 3861.99 0.000 15:5 
65 246.09 0.000 


MedBias 


188.8* 
60...1* 


0.64 
0..36* 


* if B>25%, R outside [0.5; 2] 
note: ageEnrollMiss omitted because of collinearity 


TABLE D.6 


Balancing Test Results for Propensity Score Match in Kentucky, Adult Education Students 


Unmatched 
Variable Matched 
unemployment U 
M 

collegeintent U 
M 

collegeintentmiss U 
M 

preAOPS U 
M 

preAOAE U 
M 

ageEnroll U 
M 

ageEnroll2 U 
M 

ageEnrollMiss U 
M 

female U 
M 

race black U 
M 

race _otherunknown U 
M 

singleparentAO U 
M 

singleparentAOmiss U 
M 

pellAo U 
M 

aoEnrollSem201202 U 
M 

aoEnrollSem201203 U 
M 

aoEnrollSem201301 U 
M 

aoEnrollSem201302 U 


Mean 


9.8008 8.6682 
9.6287 9.1056 


20999 36743 
#20592 2312 


-02381 2.02723 
01639 - 03063 


19048 18985 
19672 »24025 


-43651 -47296 
-45082 -4491 


29.405 28.4 
29.295 29.462 


972.48 921 17 
965.56 971...3 
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90.3 


=316.3 


-6901.6 


95:33 


83.4 


88.8 


26.9 


93.2 


80.2 


61.0 


90.2 


-40.6 


71.0 


82.7 


t-test 
p>ltl 
90 0.000 
-08 0.283 
-62 0.000 
-38 0.706 
23 0.817 
73° 0.465 
02 0.986 
82 0.413 
80 0.423 
03 0.979 
03 0.303 
13. 0.900 
73 0.463 
06 0.951 
-54 0.586 
-28 0.777 
-67 0.095 
O07 0.943 
-90 0.367 
-51 0.608 
76 0.448 
eld 10...901. 
25 0.065 
49 0.625 
-77 0.006 
-20 0.840 
-42 0.678 
.37 0.714 
-81 0.070 
42 0.673 
-32 0.188 
19 0.852 
13. 0.893 


M -04098 .0393 0.8 31.6 0.07 0.947 
aoEnrol1lSem201303 U 0873 18463 -28.6 -2.78 0.005 

M 09016 11456 =7-02 74.9 -0.63 0.532 
aoEnrollSem201401 U 19841 .08355 33.4 4.45 0.000 

M 19672 .1452 1.5:./0 58:41. L07 02287 
aoEnrollSem201402 U .02381 02275 0.7 0.08 0.938 

M 02459 .01928 3.5 -402.4 0.28 0.778 
aoEnrol1lSem201403 U 2381 11451 32.8 4.18 0.000 

M 22951 23441 =17.3 96.0 -0.09 0.928 
CIPhealth U 42857 15479 63.0 8.14 0.000 

M 42623 -38427 O37 84.7 0.67 0.506 
CIPmanu U 35714 11712 58.7 7.97 0.000 

M 34426 34557 =0.3 99.5 -0.02 0.983 
CIPanyother U - 62698 87728 -60.4 -8.15 0.000 

M . 63934 - 63637 0.7 98.8 0.05 0.962 
emptotal U i 3.0802 15:0 0.11 0.914 0.92 

M 3.0902 3.33599 =8.1. =707.9 -0.63 0.529 0.93 
earningsl U 1499.7 2056.9 =13:3 -1.38 0.168 0.76 

M 1548.9 1588.9 =L.<'0 92.8 -0.08 0.937 0.99 
earnings2 U 1524.2 2055.2 =13.0 =1635. 0.177 0.78 

M 15125 1635.4 -3.0 76.9 -0.25 0.804 1.00 
earnings3 4 U 4269.2 4112.2 lea) 0.21 0.836 1.00 

M 4040.5 4508.7 =§.6 =198.2 -0.44 0.657 0.99 
earnings5S 8 U 8844.9 7846.1 6.1 0.67 0.500 1.02 

M 8636.4 9633 26.1. 0.2 -0.47 0.638 0.90 
emp1 U -30952 41813 22267 -2.42 0.016 

M .31967 .35835 -8.1 64.4 -0.64 0.525 
emp2 U 234127 -40172 -12.5 a1235 0.076 

M - 33607 - 38034 =9 2 26.8 -0.72 0.473 
earningslsq U 1.7e+07 2.4e+07 =7 <8 -0.76 0.449 0.55* 

M 1.8e+07 1.8e+07 =0.3 96.2 -0.03 0.980 0.81 
earnings2sq U 1.7e+07 2.3e+07 -8.0 -0.81 0.416 0.70* 

M 1.7e+07 1.7e+07 =0:.5 93.2 -0.04 0.965 0.82 
earnings3 4sq U 8.7et+07 8.6e+07 0.4 0.04 0.969 0.79 

M 8.3e+07 8.8e+07 =1.8 =374. -0.14 0.887 0.89 
earningsS 8sq U 3.4e+08 3.3e+08 1.8 0.18 0.859 0.68* 

M 3.3e+08 3.8e+08 -4.4 -148.6 -0.34 0.73 0.67* 
NRS3 U 21429 .1462 Let 2.10 0.036 

M 20492 25264 -12.4 29.9 -0.89 0.377 
NRS4 U .39683 ~23797 34.6 4.06 0.000 

M 39344 34694 0.1 70.7 0.75 0.454 
NRS5 U .11905 08765 0.3 1.21 0.227 

M 12295 .08929 1.2/0 ee 0.85. 0.395 
NRS6 U 06349 .09884 =12.9 a1.31.. OvL91 

M 06557 -06371 0.7 94.7 0.06 0.953 
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NRSm U | .1746 39649 =50.6 | -5.02 0.000 

M | .18033 21523 -8.0 84.3 | -0.68 0.496 

| | 

* if variance ratio outside [0.70; 1.42] for U and [0.70; 1.43] for M 
Sample | Ps R2 LR chi2 p>chi2 MeanBias MedBias B R % 
——— 4------------- - - - - eee ee 
Unmatched | 0.267 274.99 0.000 ig ae 12.6 137.5% 0.63 
Matched | 0.033 11.29 1.000 541 4.4 41.3* 0.60 
* if B>25%, R outside [0.5; 2] 


TABLE D.7 


Balancing Test Results for Propensity Score Match in Kentucky, Developmental Education Students 


Mean Sreduct 
Treated Control Sbias |bias| 
9.1504 7.8217 44.3 
9.1504 8.9216 7.6 82.8 
01783 01527 2.0 
01783 02253 =3:.:7 -84.3 
.54376 39715 29.7 
-54376 52314 4.2 85.9 
~27229 68061 =89.6 
-27229 32185 -10.9 87.9 
29.458 28.481 93:1. 

29.458 29.484 -0.2 97.4 
991-2977 916.7 cee 
991.97 995.01 -0.4 96.0 
0 0 
0 0 

54781 56548 -3.6 

54781 52733 4.1 =15:29) 
10778 16527 -16.8 

10778 10757 0.1 99.6 
05348 05766 -1.8 

05348 05531 -0.8 56.3 
16207 1977 -9.3 

16207 16034 0:5 95:4. 
32253 27113 tl. 

32253 31432 1.8 84.0 
49676 5745 -15.6 

49676 49537 0.3 98.2 
0316 02748 2.4 

0316 03461 =128 27.0 
11507 18611 =20:.0 

11507 13263 -4.9 1Dao 
13452 14849 -4.0 


Unmatched 
Variable Matched 
unemployment U 
M 

preAOdeg U 
M 

preAOPS U 
M 

preAOAE U 
M 

ageEnroll U 
M 

ageEnroll2 U 
M 

ageEnrollMiss U 
M 

female U 
M 

race black U 
M 

race _otherunknown U 
M 

singleparentAO U 
M 

singleparentAOmiss U 
M 

pellAo U 
M 

aoEnrollSem201202 U 
M 

aoEnrollSem201203 U 
M 

aoEnrollSem201301 U 
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t=test 

t p>ltl 
20.08 0.000 
1.63 0.103 
0.70 0.484 
-0.83 0.406 
10.13 0.000 
1.03 0.305 
-29.73 0.000 
-2.70 0.007 
3.20 0.001 
-0.06 0.954 
3.37 0.001 
-0.09 0.926 
-1.21 0.228 
1.02 0.308 
-5.30 0.000 
0.02 0.987 
-0.61 0.544 
-0.20 0.842 
-3.04 0.002 
0.12 0.907 
3.90 0.000 
0.44 0.662 
-5.32 0.000 
0.07 0.945 
0.85 0.396 
-0.42 0.676 
-6.25 0.000 
-1.32 0.186 
-1.33 0.182 


BR 


M 13452 12459 2.8 28.9 0.73 0.463 
aoEnrollSem201302 U 01702 - 03168 =9.5 -2.88 0.004 

M .01702 01539 dl lt 88.9 0.32 0.749 
aoEnrollSem201303 U . 19368 17366 52 178. 90075 

M . 19368 «21125 -4.5 12:2 -1.09 0.278 
aoEnrollSem201401 U 18882 .11969 19:2 7.09 0.000 

M 18882 - 16099 Pad 59.7 1.82 0.069 
aoEnrollSem201402 U .01621 - 03168 -10.1 -3.04 0.002 

M .01621 01421 153 87.1 0.41 0.685 
aoEnrollSem201403 U »22447 14161 21.5 7.92 0.000 

M ~22447 -20593 4.8 77.6 1.12 0.262 
CIPhealth U ~54457 -20295 715.5 28.21 0.000 

M - 54457 -5188 isd 92.5 1.28 0.200 
CIPmanu U 39465 10217 719 30.95 0.000 

M .39465 42531 75 89.5 “155° ‘0..d22 
CIPanyother U 66532 Pac plop -64.4 -28.49 0.000 

M 66532 . 67639 =2.9 95.6 =0 59; “02559 
emptotal U 3.534 3.6941 -5.1 -1.73 0.083 0.99 

M 3.534 3.394 4.5 12.5 1.12 0.264 1.00 
earningsl U 1724.8 1973.9 -7.4 -2.46 0.014 0.93 

M 1724.8 1728.5 -0.1 98.5 -0.03 0.979 0-393 
earnings2 U 1894.3 1945.6 -1.4 =0.51, (0.612 1.33% 

M 1894.3 1766.6 3.5 -149.0 0.85 0.397 1.19* 
earnings3 4 U 4067.7 3880.5 2.6 0.87 0.385 0.96 

M 4067.7 3859 2.9 -11.4 0.73 0.468 1.01 
earnings5S 8 U 8050.9 6954.9 TS 2.82 0.005 1 53% 

M 8050.9 7542.2 365 53:26 0.83 0.408 120% 
emp1 U -46921 -50451 aye =2.39 0.017 

M -46921 -45218 3.4 51.8 0.85 0.396 
emp2 U -45705 -49055 -6.7 =2.27 0.023 

M 45705 43898 3.6. 46.1 0.90 0.367 
earningslsq U 1.4e+07 1.6e+07 =2.5 -0.76 0.448 0.54* 

M 1.4e+07 1.5e+07 =12 50:2 -0.34 0.73 0.75* 
earnings2sq U 1.9e+07 1.5e+07 3.8 1.90 0.058 4.28% 

M 1.9e+07 1.6e+07 331 18.5 0.76 0.449 3.438% 
earnings3 4sq U 6.8e+07 6.8e+07 =0.1 “0.01 0.989 0.03* 

M 6.8e+07 6.6e+07 0.2 =336.6 0.13 0.895 0:1 9% 
earnings5 8sq U 3.2e+08 2.2e+08 3:68 1.34 0.180 1.17% 

M 3.2e+08 2.7e+08 9 51.23 0.57 0.570 4.27* 
ACTscoreStd U =.01121 -.0224 35 1.22 0.224 1.17% 

M -.01121 -.01102 -0 98.4 -0.01 0.989 1.08 
ACTscoreStd2 U 11278 .09659 or 1.06 0.288 1.09 

M 11278 10396 17 45.5 0.40 0.688 0.94 
ACTscoreStdMissing U .8671 - 88394 =5e -1.77 0.077 

M -8671 .87545 =2.5 50.4 -0.62 0.536 
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* if variance ratio outside [0.89; 1.12] for U and [0.89; 1.12] for M 


Sample | Ps R2 LR chi2 p>chi2 MeanBias MedBias B R SVar 
—_—_— 4--------------------- ~~ - eee eee 
Unmatched | 0.389 3457.17 0.000 16.4 7.4 la a es 0.46* 71 
Matched | 0.011 38.70 0.306 3:20 229 20.9 0.57 50 


* if B>25%, R outside [0.5; 2] 


Louisiana 


TABLE D.8 


Balancing Test Results for Propensity Score Match in Louisiana, All Students 


Unmatched Mean Sreduct t-test V(T)/ 
Variable Matched Treated Control Sbias |bias| € p>ltl V(C) 
collegeintent U .09448 .09978 =17°. -3.45 0.001 Loi] 
M .09448 0937 265 85.3 0.36 0.719 1.01 
semesterl U -05455 23582 =53..2 -8.77 0.000 
M -05455 .07192 =5:. 90.4 -1.06 0.290 
semester2 U 16136 16882 =2:0 -0.39 0.696 
M 16136 13498 Ts =253'.8 1.10. 0.271 
semester3 U 10909 .06401 16. 3.47 0.001 
M -10909 -09569 4.8 70.3 0.66 0.513 
semester4 U 17045 - 26567 =23:.2 -4.30 0.000 
M -17045 - 13652 8.3 64.4 1.40 0.163 
semester5 U .02727 00332 19.6 5.93 0.000 
M “O2727 «02237 4.0 79:..5 0.47 0.641 
semester6 U .02727 .00199 2122 6.93 0.000 
M .02727 -01092 lac Pee 35:53 1.77 0.077 
semester7 U 15455 - 06998 27.0 6.11 0.000 
M 15455 . 17309 =5.9 78.1 -0.74 0.458 
semester8 U v225 11907 28.3 6.15 0.000 
M j225 -25818 -8.9 68.7 “1.15 0.251 
NRSlow U -02045 .02753 -4.6 -0.86 0.389 
M .02045 .02445 =2:.6 43.5 -0.40 0.689 
NRS3 U 05227 -07496 =9:33 -1.72 0.086 
M .05227 .06514 =5..3 43.3 -0.81 0.417 
NRS4 U -26591 18109 20.5 4.23 0.000 
M -26591 -25488 Ze 87.0 0.37 0.710 
NRS5 U 322727 »29552 -15.6 -2.96 0.003 
M ~22727 -23108 -0.9 94.4 -0.13 0.893 
NRS6 U 10909 19204 =23.3 -4.23 0.000 
M 10909 13492 =143 68.9 =1,17 0.242 
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NRSmissing 
priorged 
highschool 
gthighschool 
singleparent 
age 

agesq 

female 

white 

black 
hispanic 
otherrace 
missingrace 
pell 
unemployment 
health 
manufacturing 
othercip 
emptotal 
emp1 

emp2 

emp3 4 


emp5 8 


25-715 
25.775 


749.94 
749.94 


- 64091 
- 64091 


- 36818 
-36818 


-50227 
-50227 


2727 
.02727 


- 06364 
06364 


- 03864 
-03864 


.21591 
21591. 


6.1377 
6.1377 


- 43636 
- 43636 


235 
-35 


18409 
18409 


3.5045 
3.5045 


-48636 
-48636 


-46591 
-46591 


eO92727 
O2ILT 


56136 
-56136 


23207 
- 29084 


00829 
0175 


03284 
LS 716. 


-00796 
-01772 


06965 
13229 


25.661 
25.92 


757.8 
729.28 


-40962 
6216 


- 38043 
34536 


.591542 
-51216 


-03582 
.02729 


-03914 
-06389 


-02919 
05131 


17944 
29852 


5.876 
6.1629 


05705 
43709 


229519 
34888 


- 54328 
-18007 


2.2763 
3.6562 


-31509 
-49187 


- 30381 
-50916 


35091 
-5464 


-39502 
-55178 


61. 


-624. 


50:3 


77. 


=123% 


-163. 


91. 


-86. 


24. 


99. 


99. 


-34. 


-126. 


90. 


99. 


98. 


98. 


87. 


96. 


Ts 


89. 


94. 


-000 
-243 


~747 
149 


-000 
075 


099 
-835 


-023 
2135 


821 
676 


-813 
603 


.000 
993 


-621 
- 480 


- 606 
-770 


-360 
-998 


-O17 
- 988 


-281 
.365 


065 
.005 


-000 
742 


.000 
- 983 


-019 
~972 


-000 
877 


.000 
469 


.000 
-870 


.000 
-200 


.000 
279 


-000 
»775 


2.41* 
3452" 


1.08 
0.83* 
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earningsl U 1508.2 1153.4 3.2 2.5 0.012 0.85 
M 1508.2 1704.2 =7 53 44.8 =1.09 0.275 0.88 
earnings2 U 1458 1114.1 re 2490 0.012 0.e1* 
M 1458 1649 -7.4 44.5 tlre 0.268 0:85 
earnings3 4 U 2927 2177 4.8 208 0.005 0.837 
M 2927 3256.7 =6:..5 56.0 -1.00 0.319 0.93 
earnings5S 8 U 5491.8 4097.2 4.2 2.67 0.008 0.81* 
M 5491.8 5945.4 -4.6 67.5 -0.7 0.475 0.96 
* if variance ratio outside [0.83; 1.21] for U and [0.83; 1.21] for M 


Sample | Ps R2 LR chi2 p>chi2 MeanBias MedBias B R SVar 
——— ee ee 4-------------- ~~ - eee ee 
Unmatched | 0.321 845.85 0.000 21.7 16.1 166.1* 1.37 44 
Matched | 0.027 32:73 0.786 ees) 4.7 387% 1.00 22 


* if B>25%, R outside [0.5; 2] 
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Appendix E. Standards for Causal 
Interpretation of Impact Results 


Several criteria must be satisfied for an evaluation to be accepted as rigorous evidence of a program’s 
causal impact on participants. Although individual researchers have their own understanding of what 
constitutes scientific rigor, many organizations have developed consensus standards for judging 
whether an evaluation’s results can be interpreted causally. The consensus standards that are most 
relevant to Accelerating Opportunity are the Department of Labor’s Clearinghouse for Labor 
Evaluation and Research (CLEAR) standards and the Department of Education’s What Works 
Clearinghouse (WWC) standards. This appendix discusses how the methodology of this report meets 


these standards. 


CLEAR Standards for Causal Evidence 


CLEAR only identifies studies as having a “high” causal evidence rating if they use either the randomized 
controlled trial or the interrupted time series method. Since this report uses propensity score matching, 
it can be rated as having a “moderate” causal evidence rating. Propensity score matching studies are 
judged according to regression analysis guidelines because the analysis is conducted using a regression 
framework. The first requirement is that comparison groups are similar to treatment groups on 
preintervention characteristics (both observed and unobserved). This requirement is achieved in most 
cases after reweighting the comparison group using the results of the propensity score matching 
analysis. Some preintervention characteristics are not similar between treatment and comparison 
groups even after matching. In such cases, CLEAR evidence guidelines require that these characteristics 
are included in a subsequent regression analysis used to estimate the treatment effect. In this report, all 
matching variables are included in the regression that produces the impact estimates. All impact 
estimates provided in this report are estimated using a regression adjustment of the matched 


comparison group, so all impact estimates in the report meet this standard. 


Different types of interventions require controlling for different pre-intervention characteristics. 
For this report, we followed the protocol for evaluations of community college programs. This protocol 
requires the inclusion of the following control variables: age, race or ethnicity, gender, state, a 


preintervention measure of financial disadvantage, and a preintervention measure of academic 
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achievement. Each of these controls is included in the analysis. Pell grant receipt and preintervention 
earnings are included as measures of financial disadvantage. NRS levels, and standardized 
postsecondary placement tests (including ACT, SAT, ACCUPLACER, and COMPASS, depending on the 
state), high school completion status, and prior postsecondary experience are included as measures of 
preintervention academic achievement. Because employment and earnings are key outcomes, eight 
quarters of lagged employment and earnings are included as matching variables as well. Lagged 
outcomes are not required in the CLEAR community college review protocol, but they are 
recommended in the causal evidence guidelines. Although PSM is not a method that identifies the 
effects of an intervention by comparing changes in an outcome over time (as in a difference-in- 
differences or fixed effects model), equivalent trends are guaranteed by matching on prior trends in 


employment and earnings and controlling for these variables in the regression. 


An important concern in any propensity score matching study is the equivalence of the comparison 
group on unobserved characteristics. This study avoids this problem by matching on and controlling for 
pretreatment employment and earnings histories. Although these characteristics are “observed,” they 
are determined by several unobserved characteristics that are also determinants of the outcomes. 
Controlling for pretreatment employment and earnings essentially controls for any idiosyncratic 
individual characteristics affecting labor market performance that are not observed in the analysis. We 
also control for semester of enrollment, which should absorb unobserved characteristics associated 
with differences between fall enrollments (which may disproportionately be coming from high school) 


and other enrollees. 


The second requirement is that the study does not have any confounding factors. The analysis does 
not suffer from confounding factors, including N=1 confounds. Students in the comparison group attend 
the same schools and live in the same states as treatment cases. Although in some cases very few 
treatment cases have a certain value for a control variable (e.g., very few, if any, treatment cases had 
prior post-secondary experience), the comparison group was sufficiently large relative to the treatment 
group that there are always comparison cases with the same characteristics. These N=1 confounds are 
automatically reported by the statistical software, and all statistical output was checked to ensure the 


absence of N=1 confounds. 


The third requirement is that participants do not anticipate the intervention. There is no risk that 


students anticipated the AO intervention or adjusted their behavior to gain entry into the program. 


The analysis estimates (average) individual level effects, not group level effects such as the impact 


of AO onaschool or acounty. As a result, the fourth requirement, which relates to group compositional 
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changes, does not apply to this study. The analysis also does not use fixed effects, random effects, or 
instrumental variables, so requirements associated with those designs do not apply to this study. Our 
measure of the predicted probability of college is a predicted value estimated in a prior model, but it is 
not functioning as an instrument and no exclusion restrictions are satisfied (i.e., the variables in the 


model determining the predicted probability of attending college are all included in the main model). 


WWC Standards for Causal Evidence 


Similar to CLEAR, WWC only identifies studies as meeting WWC standards “without reservations” if 
they use a randomized controlled trial. Because this evaluation uses propensity score matching, it can 
be rated as meeting WWC standards “with reservations,” which is available to well-executed quasi- 
experimental designs. The first WWC requirement for studies that do not randomly assign treatment is 
that baseline equivalence is achieved between the treatment and comparison group. Baseline 
equivalence is satisfied when the mean differences between two individual characteristics is less than 
0.05 standard deviations. Although some baseline characteristics satisfy this criteria (see appendix D), 
many do not. In studies with many matching variables, it is common that no vector of weights can 
simultaneously satisfy baseline equivalence on all variables. Rather than build in a bias in favor of 
studies that use fewer matching variables to achieve greater baseline equivalence, postmatching 
regression adjustment can satisfy baseline equivalence. All impact estimates in this analysis use 


regression adjustment to control for any remaining lack of baseline equivalence. 


WWC also requires that outcome measures have face validity and reliability. All the outcome 
measures in this analysis satisfy those criteria. The outcome measures do not attempt to measure any 
latent characteristic—they are direct measures of credits earned, credentials earned, employment, and 
earnings. Outcome measures must also not be overaligned with the treatment, and they must be 
collected in the same manner for both the treatment and comparison groups. Our analysis satisfies both 
requirements. The AO treatment is not defined by the accumulation of a specific number of credits or 
credentials, much less subsequent employment, so the outcome measure is not overaligned. Outcomes 
are measured for treatment and comparison cases using the same data sources; postsecondary 
administrative records for credits and credentials, and unemployment insurance wage records for 


employment and earnings. 


WWC requires that if subgroups are analyzed findings are also presented for the full sample. This 


analysis satisfies that requirement. Recruitment source subgroups in Kansas and Kentucky are 
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analyzed, but analyses of the total sample in these states are also included. Finally, WWC requires that 
there are no confounding factors associated with treatment. As noted in the CLEAR discussion above, 


there are no relevant confounding factors in this analysis. 
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